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Student-Level Analysis of Year 1 (2003 - 2004) Achievement 
Outcomes for Tennessee Charter Schools 

Abstract 

This report presents student-level achievement results for the four charter 
schools that began operation in Tennessee during the 2003-04 academic year. To 
conduct a rigorous and valid analysis of student achievement outcomes at these 
schools, we employed a matched program-control design at the student level, whereby 
each charter school student was paired to a comparable “control” student attending the 
same or similar school during 2002-03. The four charter schools were: Circles of 

Success in Learning Academy (COSLA), Memphis Academy of Health Sciences 
(MAHS), Memphis Academy of Science and Engineering (MASE), and the Smithson- 
Craighead Academy (SCA). The first three schools are located in Memphis and the 
fourth in Nashville. Participants in the study were 14 second graders from COSLA, 70 
sixth graders from MAHS, 134 seventh graders from MASE, and 9 fourth graders from 
SCA. All had scores on the spring, 2003 (pre-implementation or “pretest”) and the 
spring, 2004 (posttest) Tennessee Comprehensive Assessment Program/Achievement 
Test (TCAP/AT) in Reading, Language Arts, and Math. 

The analyses of 2004 TCAP/AT scores for the three Memphis cohort schools 
showed directional advantages for the charter school students over their matched 
control counterparts on all three subtests. However, only the effect for MASE in Math 
was statistically significant. The median effect sizes were +0.15 for test scores 
unadjusted for pretest differences and +0.12 for adjusted scores, indicating small to 
moderate effects. When the results for all three schools were combined, significant 
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advantages for the charter school students occurred on both Reading ( ES = +0.19) and 
Math (ES = +0.19), with a directional advantage in Language Arts (ES = +0.12). 
Attendance was also significantly higher at two of the Memphis schools and directionally 
higher at the third. For SCA, however, the comparisons to control students yielded 
comparable to negative results. Because only 9 student matches were available for 
SCA, these results need to be viewed very cautiously relative to those from the three 
schools and much larger sample sizes in the Memphis cohort. 

Overall, the first-year achievement outcomes seem highly suggestive and 
impressive in view of the: (a) early (first-year) implementation of the charter school 
programs; (b) charter school students’ transition to new schools, a factor normally 
associated with lower achievement; and (c) comparable magnitude of the effect sizes in 
Memphis to those obtained in prior research on established comprehensive school 
reform (CSR) models. The present study will be replicated over the next several years 
to examine first-year results for newly established charter schools and longitudinal 
outcomes for the entire population of Tennessee charter schools. 
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Student-Level Analysis of Year 1 (2003 - 2004) Achievement 
Outcomes for Tennessee Charter Schools 



This report presents student-level achievement results for the four charter 
schools that began operation in the State of Tennessee in the 2003-04 academic year. 
To conduct a rigorous and valid analysis of student achievement outcomes at these 
schools, we employed a matched program-control design at the student level, whereby 
each charter school student was paired to a comparable “control” student attending the 
same or similar school during 2002-03. Such a design has the critical advantages of 
controlling for pre-program achievement and other relevant student and school 
variables. 

Description of the Charter Schools 

Three of the charter schools established in 2003-04 were located in Memphis. Their 
characteristics are described briefly below. 

Circles of Success in Learning Academy (COS LA). COSLA is an urban school 
located in Memphis, Tennessee. In the first year, 60 African American students were 
enrolled, with 20 students each, in one classroom per grade, in grades K, 1, and 2. Of 
the 60 students, 66.6% receive free or reduced-price lunch. As a literacy-focused 
school using the Success for All reading program, COSLA emphasizes the integration 
of reading and writing activities across all curricular areas. 

Memphis Academy of Health Sciences (MAHS). MAHS is located in a high- 
poverty area north of downtown Memphis, Tennessee. There are 99 students attending 
grade 6, the only grade served this first year. African American students represent 
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100% of the school’s population; 77% of the students receive free or reduced-price 
lunches. The curriculum developed and employed at MAHS is a standards-based, 
interdisciplinary program that incorporates projects and experiential learning centered 
on a health sciences theme. 

Memphis Academy of Science and Engineering (MASE). MASE is located in 
downtown Memphis in an urban medical center district surrounded by the University of 
Tennessee Health Science Center. For the first year, the school opened with 147 
seventh-grade students of which 97.9% were African American, 1.4% Caucasian, and 
0.7% Latino. Nearly two-thirds (65% or 96 students) qualified for free or reduced-price 
lunch. The main focus of the school for this first year was to foster a sense of school 
family/community, improve the Language Arts skills of the students, and to create a 
culture of academic excellence. The goal of the principal and primary stakeholder is to 
make a showcase charter school that created the science and math leaders of 
tomorrow. 

The fourth charter school was located in Nashville, and is described below. 

Smithson-Craighead Academy (SCA). This urban charter school is located in 
the city of Nashville, Tennessee. The mission of the school is to provide basic 
academic skills necessary to prepare students for success throughout their K-12 
academic education. The student population consists of approximately 155 
kindergarten through fourth grade students who previously attended schools designated 
“at-risk.” The student population is 99% African-American. 
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Method 



Participants in the study were 14 second graders from COSLA, 70 sixth graders 
from MAHS, 134 seventh graders from MASE, and 9 fourth graders from SCA All had 
scores on the spring, 2003 (pre-implementation or “pretest”) and the spring, 2004 
(posttest) Tennessee Comprehensive Assessment Program/Achievement Test 
(TCAP/AT) in Reading, Language Arts, and Math. Normal Curve Equivalent (NCE) 
scores, a standard score distribution with a mean of 50, were used in all analyses. 

For the Memphis cohort, each charter school student was matched with a control 
student based on eight criteria: 2002-03 enrollment (at the same school as the charter 
school counterpart), grade level, race, gender, free/reduced price lunch status, and the 
2003 subtest scores on TCAP/AT in Reading, Language Arts, and Math. When an 
appropriate match for a charter school student could not be found from his/her former 
(2002-03) school, the closest match from one of the schools formerly attended by 
his/her present classmates was selected using all other criteria. 

For SCA, there were only 21 students who took the TCAP/AT in 2004. Of these, 
only 14 had pretest (2003) scores. This subsample was further reduced to 9 by 
applying the above control-student matching procedure, excluding the gender variable; 
and to only 5 if gender were included. It was therefore decided to use the former, larger 
subsample to permit an exploratory inferential (comparative) statistical analysis. 
Because of the small sample size, however, and unique conditions in the schools 
relationship to their respective parent districts and community sponsors in the two cities, 
we present the Memphis and Nashville results in separate sections below. 
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Memphis Results 

Pre-Implementation 



Descriptive Results 

Descriptive statistics for pre-implementation measures (2003) are presented in 
Table 1. In the year prior to enrollment in the charter schools, students in COSLA and 
MAHS, as well as their respective matched-control samples, scored below the national 
norm (50) on all three subtests (Reading, Language Arts, and Mathematics). MASE 
students and their matched controls had mean NCE scores slightly below the national 
norm on the Mathematics subtest but higher than the national norm on the Reading and 
Language Arts subtests. As can be seen in the table, the effect sizes 1 (ES) comparing 
charter and control pretest means were extremely small, suggesting a high degree of 
comparability. 



1 Effect sizes indicate the number of standard deviations by which the “program” or experimental group 
surpasses the control group. An ES > 0.25 would be considered to be strong and educationally 
meaningful. 
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Table 1 



Descriptive NCE statistics for Charter School Students and Control Students on the 
2002-2003 TCAP/AT 




Circles of Success N=14 


GRADE 1 Mean 


45.57 


45.07 


35.00 




Standard Deviation 


21.17 


20.56 


16.59 


Control Students N=14 


GRADE 1 Mean 


44.43 


42.36 


36.14 




Standard Deviation 


20.87 


20.85 


15.28 




Effect Size 


+0.05 


+0.13 


-0.07 


Memphis Academy of Health Sciences N=70 










GRADE 5 Mean 


42.14 


43.76 


40.14 




Standard Deviation 


17.64 


19.02 


15.20 


Control Students N=70 


GRADE 5 Mean 


42.21 


43.00 


40.41 




Standard Deviation 


17.09 


18.28 


15.37 




Effect Size 


0.00 


+0.04 


-0.02 


Memphis Academy of Science & Engineering N= 


134 








GRADE 6 Mean 


57.58 


56.15 


49.25 




Standard Deviation 


16.64 


15.81 


15.81 


Control Students N=134 


GRADE 6 Mean 


57.22 


56.19 


49.71 




Standard Deviation 


16.70 


14.86 


15.56 




Effect Size 


+0.02 


+0.00 


-0.03 


Note: The means represent scores in the year and grade prior to students’ charter school enrollment. 



Inferential Results 

Separate one-way ANOVAs to determine charter school and control group 
equivalence revealed non-significant results for all three subtests (Reading, Language 
Arts, and Mathematics). Thus, there is no basis for inferring that charter and control 
students differed in achievement in the grade prior to charter school enrollment. 

Post-Implementation 

Correlational Results 

Correlations between the pre- and post-implementation measures for COSLA 
were at least close to moderate in strength. For Reading, r = +.43 (p<. 05); for 
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Language Arts r= +.47 (p<. 05); and for Math, r= +.61 (pc.01). Intercorrelations among 
the pre- and post-implementation measures ranged from +.36 to +.61 . 

Pre- and post-implementation correlations for MAHS were all statistically 
significant and moderate to strong in magnitude: Reading, r= +.60 (pc.01); Language 
Arts, 

r = +.75 (pc.01); and Math, r= +.81 {p<. 01). Intercorrelations ranged from +.58 to +.81 . 

MASE also had moderate to strong pre- and post-implementation correlations, 
which were statistically significant: Reading, r= +.72 (pc.01); Language Arts, r= +.69 
(pc.01 ); and Math, r = .81 (pc.01 ). Intercorrelations ranged from +.61 to +.81 . 
Descriptive Results 

Descriptive statistics for post-implementation measures are presented in Table 2. 
COSLA ( M = 48.07) and MAHS ( M = 46.46) students scored close to the national norm 
of 50 in Language Arts, whereas MASE students approximated or exceeded the 
national norm on all subtests. Inspection of the unadjusted means shows that the three 
charter schools directionally surpassed the control students on all three subtests, with 
ESs ranging from +0.03 to +0.69. 
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Table 2 



Descriptive NCE statistics for Charter School Students and Control Students on the 
2003-2004 TCAP/AT 




Circles of Success N=14 



Mean 


40.29 


48.07 


36.64 


Adjusted Mean 


39.68 


47.66 


36.21 


Standard Deviation 


12.56 


12.56 


17.07 


Control Students N=14 








Mean 


37.29 


38.64 


36.29 


Adjusted Mean 


37.89 


39.05 


36.70 


Standard Deviation 


19.86 


13.67 


13.59 


Effect Size 


+0.15 


+0.69 


+0.03 


Adjusted Effect Size 


+0.09 


+0.63 


-0.04 


Memphis Academy of Health Sciences N=70 








Mean 


43.47 


46.46 


39.81 


Adjusted Mean 


43.42 


46.31 


39.82 


Standard Deviation 


13.75 


16.61 


15.79 


Control Students N=70 








Mean 


40.36 


42.91 


38.03 


Adjusted Mean 


40.40 


43.06 


38.03 


Standard Deviation 


15.34 


17.97 


16.91 


Effect Size 


+0.20 


+0.20 


+0.11 


Adjusted Effect Size 


+0.20 


+0.18 


+0.11 


Memphis Academy of Science & Engineering N= 


134 






Mean 


51.50 


55.43 


48.71 


Adjusted Mean 


51.49 


55.47 


48.84 


Standard Deviation 


17.00 


20.42 


15.65 


Control Students N=134 








Mean 


49.66 


54.49 


45.81 


Adjusted Mean 


49.67 


54.46 


45.68 


Standard Deviation 


15.82 


16.97 


15.45 


Effect Size 


+0.12 


+0.06 


+0.19 


Adjusted Effect Size 


+0.12 


+0.06 


+0.20 



Inferential Analyses 

A multivariate analysis of covariance (MANCOVA) was used to assess program 
effects. Each charter school and its matched control group were compared on the 2004 
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TCAP/AT Reading, Language Arts, and Mathematics subtests. The 2003 (pretest) 
TCAP/AT subtests were used as covariates. 

Circles of Success Learning Academy - Grade 2. The 2003 Language Arts 
subtest was the only significant covariate in the MANCOVA (p<. 02). The multivariate 
effect of Program did not reach significance, F( 3,23)=2.32, p=.10, eta 2 =.232. Thus, the 
COSLA and matched control means were not significantly different on the various 
subtests. In interpreting these results, it is important to consider that the very small 
sample size of 14 matched pairs limits the power of the analysis for detecting program 
effects. For example, the directional superiority of COSLA in Language Arts (ES = 
+0.69) is suggestive of an educational impact that might prove highly meaningful and 
significant with a larger sample. 

Memphis Academy of Health Sciences - Grade 6. The Reading, Language Arts, 
and Math covariates were all highly significant in the MANCOVA (all p’ s<.02). The 
multivariate effect of Program, however, did not reach significance, F(3,133)=1 .15, 
p=.331, eta 2 =. 025. Thus, despite the directional superiority of the MAHS over the 
matched control students on all three subtests, the effects were not sufficiently large to 
indicate reliable differences between groups. 

Memphis Academy of Science and Engineering - Grade 7. The three covariates 
were all highly significant in the MANCOVA (all p’s c.000). Of note, the multivariate 
effect of Program was also significant, F(3,261)=3.10, p=. 027, eta 2 =. 027. Univariate 
tests (ANCOVA) were conducted on each of the dependent measures (Reading, 
Language Arts, and Math subtests). The univariate results were significant for the Math 
subtest only, F(1 ,263)=8.89, p=. 003, eta 2 =. 033. As shown in Table 2, the directional 
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superiority of MASE students over the control students on the Reading and Language 
Arts subtests was not found to be statistically significant. 

Aggregate Memphis Results 

Supplementary analyses examined the combined results for the three schools. 
Although combining the results offers the advantage of increased sample size and 
higher statistical power, it has the disadvantage of treating as psychometricaliy 
equivalent NCE scores from varied charter school programs and especially, from 
different grade levels directly confounded with (specific to) the programs being studied. 

Pre-Implementation 

Descriptive statistics for pre-implementation measures are presented in Table 3. 
In the year prior to enrollment in the charter schools, the charter students and the 
matched control students scored above the national norm (50) on two out of three 
subtests (e.g., Reading and Language Arts). 



Table 3 

Aggregate Descriptive NCE statistics for Charter School Students and Control Students 
on the 2002-2003 TCAP/AT 




Charter Schools N=218 


Mean 


51.85 


51.46 


45.41 




Standard Deviation 


18.67 


18.13 


16.38 


Control Students N=218 


Mean 


51.58 


51.06 


45.85 




Standard Deviation 


18.47 


17.60 


16.19 




Effect Size 


+0.01 


+0.02 


-0.03 
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Separate one-way ANOVAs to determine charter school and control group 
equivalence revealed non-significant results for all three subtests (Reading, Language 
Arts, and Mathematics). As indicated by a visual comparison of means and the near- 
zero effect sizes, the charter school group performed similarly to matched control group 
on the pre-implementation measures 

Post-Implementation 

Descriptive statistics for post-implementation measures are presented in Table 4. 
As can be seen, the charter school group demonstrated a directional advantage over 
the control group on all three subtests. 



Table 4 

Aggregate descriptive NCE statistics for Charter School Students and Control Students 
on the 2003-2004 TCAP/AT 




Mean 


48.20 


52.08 


45.08 


Adjusted Mean 


48.19 


52.03 


45.19 


Standard Deviation 


16.27 


19.26 


16.39 


Control Students N=218 








Mean 


45.88 


49.76 


42.70 


Adjusted Mean 


45.00 


49.81 


42.59 


Standard Deviation 


16.59 


18.09 


16.25 


Effect Size 


+0.14 


+0.13 


+0.15 


Adjusted Effect Size 


+0.19 


+0.12 


+0.16 



A multivariate analysis of covariance (MANCOVA) was used to assess program 
effects. All three covariates were highly significant in the MANCOVA (all p’s <.001 ). 
The multivariate effect of Program was also significant, F(3,429)=3.22, p=. 023, 
eta 2 =. 022. Univariate tests (ANCOVA) were conducted on each of the dependent 
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measures (Reading, Language Arts, and Math subtests). The univariate results were 
significant for the Reading subtest, F(1 ,431)=4.68, p=.031, eta 2 =. Oil, and for the Math 
subtest, F(1 ,431)=8.65,p=.003, eta 2 =. 020. The univariate results for the Language Arts 
subtests were not significant, but approximated the .05 alpha level, F(1,431)=3.56, 
p=. 060, eta 2 =. 008. As shown in Table 4, weak to moderate adjusted mean effect sizes, 
ranging from +0.12 (Language Arts) to +0.19 (Reading), were obtained. 

Memphis Attendance Results 

Supplemental analyses examined attendance (absences) rates for students at 
the three Memphis schools compared to their matched control student counterparts. In 
two of the three comparisons, results were significant and reflecting lower absence 
rates in the charter schools. Specifically, large and significant advantages were shown 
for both MAHS ( M abs = 4.33% vs. 6.85% for controls), /(1 36)= 2.46, p = .015, ES = 

+0.34; and MASE ( M abs = 3.13% vs. 4.74% for controls), f(266)= 2.71, p= .007, ES = 
+0.274 Noting the much smaller sample sizes involved, a nonsignificant directional 
advantage was indicated for COSLA, ( M abs = 3.56% vs. 6.1 1% for controls), f(26)= 1 .99, 
p= .057, ES = +0.61 . 

Nashville Results 

Both pretest (2002-03) and posttest (2003-04) results for SCA are summarized in 
Table 5. The table also provides the results of t tests conducted to compare inferential^ 
the SCA and control student means. As shown on the table, both groups scored nearly 
identically in the pre-implementation year, thus verifying the accuracy of the matched- 
pair procedure. On the posttest, SCA had a lower mean than the control group in 
Reading (ES = -.12) and especially in Mathematics (ES = -0.54). Neither of these 
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differences was statistically significant, however. Because there were only 9 matched 
pairs involved in the analyses and gender could not be employed as a matching 
variable, these results need to be viewed with a high degree of caution. 
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Table 5 



Descriptive and inferential NCE statistics for Smithson-Craighead Academy (SCA) Students and Control 
Students on the 2002-03 and 2003-2004 TCAP/AT 



Normal Curve Equivalent Score (NCEs) 




Smithson Craighead 


02-03 


3 


Mean 


43.11 


45.67 


47.56 


(n = 9) 






Standard Deviation 


8.51 


8.70 


12.60 


Control (Matched 
Case) 


02-03 


3 


Mean 


42.56 


45.22 


46.78 


(n = 9) 






Standard Deviation 


8.81 


9.82 


11.19 








t - value (p) 


.14 (.893) 


.10 (.920) 


.14 (.892) 








Effect Size 


+0.06 


+0.05 


+0.07 


Smithson Craighead 


03-04 


4 


Mean 


40.22 


45.00 


35.00 


(n = 9) 






Standard Deviation 


9.14 


9.12 


12.39 


Control (Matched 
Case) 


03-04 


4 


Mean 


40.22 


46.56 


41.33 


(n = 9) 






Standard Deviation 


8.90 


16.89 


11.18 








t - value (p) 


0.00 (1.000) 


-.24 (.811) 


-1.14 (.272) 








Effect Size 


0.000 


-0.12 


-0.54 



Conclusions 

Results from the first-year analyses of student achievement in the three Memphis 
charter schools should be interpreted as positive overall. Specifically, means on all 
three subtests of the TCAP-AT directionally favored students from each of the three 
charter schools over matched-control counterparts attending similar district schools. 
Statistically significant advantages were obtained for MASE students in Math and the 
combined charter school (three-school) sample in Math and Reading. Attendance was 
also significantly higher at two of the schools and directionally higher at the third. The 
encouraging findings for the Memphis cohort, however, were not replicated in the 
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exploratory analysis of the 9 available matched pairs for the Nashville school (SCA). It 
is expected that larger sample sizes in the Year 2 study (2004-05) will permit a more 
meaningful analysis of SCA’s level of success in raising student achievement. 

It is important to note that compared to prior research (Bulkley and Fisler, 2002; 
Greene, Forster, & Winters, 2003), particularly the recent highly criticized study by the 
American Federation of Teachers (Page, 2004), our design provided rigorous control 
over the students to which the charter school samples were compared. Specifically, 
each control student was one who relative to his/her charter school counterpart: (a) 
either attended the same school or a highly similar school during the prior year; (b) was 
of the same race, gender, and poverty status; and (c) scored similarly on all three 
TCAP/AT subtests prior to charter school enrollment. The high degree of program- 
control group comparability helps to eliminate sampling bias as a threat to the validity of 
the findings. 

The positive trends in the first-year Memphis achievement outcomes are certainly 
encouraging. Still, we recommend that caution be exercised before reaching certain 
conclusions. First, without adjusting for pretest scores, the median effect size out of the 
nine obtained (3 schools x 3 subtests) is +0.15, indicating a weak to moderate effect 
(Cohen, 1988). With the pretest adjustment, the median falls slightly to +0.12. Second, 
out of the 12 inferential posttest (2004) comparisons (the above 9 plus the 3 aggregate 
analyses), only 3 were statistically significant: Math for MASE ( ES = +0.20) and 

Reading [ES = +0.19) and Math [ES = +0.16) for the combined schools. Lack of 
significance indicates an unacceptable probability (> 5%) that the differences obtained 
may be due to chance. A third consideration is that, even though the charter and 
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control samples are highly similar demographically, the former might still have the 
advantage, manifested through the school “choice” option, of families that are more 
involved in, and perhaps more knowledgeable about, their children’s education. A 
fourth consideration is that the positive trends were not evidenced for the Nashville 
charter school, although the serious limitations of having only 9 matched SCA-control 
group pairs has been noted throughout this report. 

Given the above caveats, it seems fair to present as a counterpoint the more 
liberal view that even the modest advantages obtained for the three charter schools 
reflect substantive attainment and promise for future success. First, a widely held belief 
by educational reform experts and researchers is that school change takes several 
years to manifest itself in observable outcomes (Fullan, 2000; Sizer, 1992; Levin, 1993). 
Imagine restructuring a business by bringing together in a new facility, under different 
policies, strategies, and conditions, totally new management and employees who have 
never worked together before. Would higher profits be expected in the short-run than 
obtained the year before? As Desimone (2002) indicates in a review of research on 
CSR, “The slow pace of school reform affects the ability to assess implementation 
success, as well as the ability to measure effects on students, teachers, and parents” 
(p. 455). Adopting new school programs — let alone creating an entirely new school — 
could, due to newness and disruption, actually lower rather than raise student 
achievement in the first few years (see Ross et al., 2001). Second, research on school 
transitions performed specifically with the TCAP system (Sanders & Horn, 1995a, 
1995b) and in other contexts (see Larson, Moneta, Richards, & Wilson, 2002; Larson & 
Richards, 2000), shows decreases in student achievement associated with changing 



18 




schools. In this regard, the entire charter school sample changed schools from 2002-03 
to 2003-04, whereas only 14% of the COSLA and 54% of the MASE matched control 
students changed schools. Nearly all (91%) of the MAHS matched controls, however, 
changed schools in graduating from elementary to middle school. Third, although the 
median effect sizes for the individual charter students were modest in absolute size (i.e., 
from +0.12 to +0.15), it should be considered that in a recent meta-analytic study of 29 
Comprehensive School Reform (CSR) models, Borman et al. (2003) found an overall 
effect size of from +0.10 to +0.14, with the range for the “most successful” category 
being +0.17 to +0.21. Only 3 out of the 29 models achieved this high status (Direct 
Instruction, School Development Program, and Success For All). In this regard, it 
seems quite impressive that newly opened schools mirrored the effects of well- 
established CSR models by raising, on the average, each enrollee over a tenth of a 
standard deviation compared to control students. Simply put, raising an entire school 
population by .10 SD would be of huge benefit both educationally and economically. 

Considering the above factors, even schools with strong potential for success 
might demonstrate no effects or even negative outcomes in their first year. In contrast, 
positive patterns were evidenced here. The true test of school effectiveness, however, 
is not what happens in a given year, but what is sustained over time (Snipes & 
Casserly, 2004). Accordingly, the present study will be replicated over the next several 
years to examine first-year results for newly established charter schools and longitudinal 
outcomes for the entire population of all Tennessee charter schools. 
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